Exact Representations from Feed-Forward Networks

We present an algorithm to extract representations from multiple hidden layer, multiple output feedforward perceptron threshold networks. The representation is based on polytopic decision regions in the input space{ and is exact not an approximation like most other network analysis methods. Multiple examples show some of the knowledge that can be extracted from networks by using this algorithm, including the geometrical form of artifacts and bad generalization. We compare threshold and sigmoidal networks with respect to the expressiveness of their decision regions, and also prove lower bounds for any algorithm which extracts decision regions from arbitrary neural networks.